Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

changed load balancer ip #207

Merged
merged 1 commit into from
Apr 5, 2019
Merged

Conversation

scottyhq
Copy link
Member

@scottyhq scottyhq commented Apr 5, 2019

had to re-deploy production cluster manually because it seems that circleci failing due to a timeout on the first deployment lead to a helm list STATUS=FAILED. apparently this condition leads to a number of issues (helm/helm#3353 (comment)). So we solved this pickle by removing the failed jupyterhub and re-deploying locally:

  1. remove failed cluster
helm delete icesat2-prod --purge
kubectl delete namespace icesat2-prod
  1. comment out this file
    https://github.com/pangeo-data/pangeo-cloud-federation/blob/prod/deployments/icesat2/config/prod.yaml

  2. initial helm deploy (run from local pangeo-cloud-federation directory)

RELEASE=icesat2
helm upgrade --install --namespace $RELEASE-prod $RELEASE-prod pangeo-deploy -f deployments/$RELEASE/config/common.yaml -f deployments/$RELEASE/config/prod.yaml -f deployments/$RELEASE/secrets/prod.yaml --set pangeo.jupyterhub.singleuser.image.tag=2035c99 --set pangeo.jupyterhub.singleuser.image.name=783380859522.dkr.ecr.us-west-2.amazonaws.com/pangeo --timeout 3000
  1. added in loadBalancerIP (from kubectl --namespace=icesat2-prod get svc) to
    https://github.com/pangeo-data/pangeo-cloud-federation/blob/prod/deployments/icesat2/config/prod.yaml

  2. re-run helm command from step 3)

RELEASE=icesat2
helm upgrade --install --namespace $RELEASE-prod $RELEASE-prod pangeo-deploy -f deployments/$RELEASE/config/common.yaml -f deployments/$RELEASE/config/prod.yaml -f deployments/$RELEASE/secrets/prod.yaml --set pangeo.jupyterhub.singleuser.image.tag=2035c99 --set pangeo.jupyterhub.singleuser.image.name=783380859522.dkr.ecr.us-west-2.amazonaws.com/pangeo --timeout 3000

this might prevent the situation from reoccurring in the future:
berkeley-dsep-infra/hubploy#17

@jhamman jhamman merged commit 60e44c9 into pangeo-data:staging Apr 5, 2019
@scottyhq scottyhq deleted the icesat2-redeploy branch May 14, 2019 17:25
scottyhq pushed a commit to scottyhq/pangeo-cloud-federation that referenced this pull request Jul 12, 2019
TomAugspurger pushed a commit to TomAugspurger/pangeo-cloud-federation that referenced this pull request Apr 23, 2020
* Add Skaffold configuration and readme

* Move skaffold-skaffolding to skaffold subfolder

Co-authored-by: Alex <[email protected]>

[ci skip]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants